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Background / Context: 

Description of prior research and its intellectual context. 

Recent research on dropout prevention indicates that the ninth grade is the “make or break year” 
for students to be on track to graduate (Allensworth & Easton, 2007; Herlihy, 2007; Kennedy & 
Monrad, 2007). More students fail ninth grade than any other grade in high school, and a 
disproportionate number of students who are held back in ninth grade subsequently drop out 
(Herlihy, 2007). The transition into a larger school system with a different structure, new 
teachers and peers, and more challenging coursework is often associated with poor grades in core 
subjects, decline in student achievement, failure to be promoted to the next grade, student 
disengagement in the classroom, behavioral problems, and decline in student motivation 
(Alspaugh, 1998; Herlihy, 2007; Isakson & Jarvis, 1999; Kennelly & Monrad, 2007). 

In addition to the academic demands of high school, students’ report facing a variety of new non- 
academic challenges upon their arrival in high school. Researchers Barber and Olsen (2004) 
found that ninth graders perceive less support and monitoring from teachers and principals and 
generally like school less than they did in middle school. In another study, middle school 
students identified academic ability as especially important to their success in secondary school; 
however, after they entered high school, students identified additional components essential to 
success in school, including time management, ability to stay on task, social skills, and behavior 
(Zeedyk, Gallacher, Henderson, Hope, Husband, & Lindsay, 2003). In this same study, students 
reported that social concerns, such as bullying and establishing peer relationships at secondary 
school, often overshadowed concerns about academics. Similarly, Akos and Galassi (2004) 
found that students’ greatest concerns about starting high school revolved around the amount of 
homework, class difficulty, and organizational issues (e.g., getting lost). 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

To provide greater support for students as they transition to ninth grade, the Texas Legislature 
funded the Texas Ninth Grade Transition and Intervention (TNGTI) grant program. TNGTI 
grantees implemented a variety of supports to at-risk students transitioning to ninth grade, 
including a summer transition program, an early warning data system, and research-based 
interventions targeted to meet student needs. The study examined the effectiveness of the 
supports schools provide under the TNGTI grant program. Specifically, the study compared the 
performance of students who participated in the TNGTI summer transition program to similar 
students who did not participate using a propensity score stratification approach. Students’ ninth 
grade attendance and performance on math and reading Texas Assessment of Knowledge and 
Skill (TAKS) assessments were the outcomes of interest. In addition, we examined the theory 
that the TNGTI program would improve student academic performance by increasing their ninth 
grade attendance by examining whether ninth grade attendance was a mediator for academic 
achievement. 
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Setting: 

Description of the research location. 

School districts and open-enrollment charter schools from across the state of Texas were eligible 
to apply for the 2009-10 TGNTI grant program if they met all of the following criteria: (1) 
served 75 percent or more economically disadvantaged students in each of the 2006, 2007, and 
2008 school years; (2) had a population of at least 25 eighth graders in 2007-2008, and (3) had a 
ninth grade retention rate for 2007-2008 that placed them in the highest three school districts or 
open-enrollment charter schools within a comparable size category. Campuses selected to 
participate in the program must have served at least 50 percent at-risk students in the 2007-2008 
school year. In the 2009-10 school year, 24 districts applied for and received TNGTI grants, 
serving a total of 63 school sites throughout the state. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features or characteristics. 

The analytic sample consisted of 30,735 ninth graders across the 63 TNGTI campuses. Of these 
students, 3,013 attended the TNGTI summer program. 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 

To provide greater support for students as they transition to ninth grade, the Texas Legislature 
funded Texas Ninth Grade Transition and Intervention (TNGTI) grant program. TNGTI grants 
provide funding for eligible districts and open-enrollment charter schools to implement three 
supports for identified students: (1) a summer transition program to introduce incoming ninth 
graders to high school culture and develop their academic, social, and study skills; (2) an early 
warning data system to monitor the progress of program participants throughout the school year; 
and (3) fall and spring interventions for participants who exhibit warning indicators for retention 
or drop out. These supports consist of successful ninth grade transition and intervention 
programs identified in the literature on dropout prevention (Heppen & Bowles Therriault, 2008). 

Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

The study employed a quasi-experimental research design to examine the effect of participating 
in the TNGTI summer transition program. Specifically, the study compared the performance of 
students who participated in the TNGTI program to similar students who did not participate 
using a propensity score stratification approach. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

In any evaluation of a program where participants are not randomly assigned to participate in the 
program or not, the problem of selection is paramount. We know that students who participate in 
the TNGTI program are different from those who do not attend above and beyond differences in 
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that arise as a result of the program. These differences can bias estimates of program 
effectiveness because they make it difficult to disentangle pre-existing differences between 
students who attended the program and those who did not from the effect of attending the 
program. 

Propensity score matching is a two stage process designed to address this problem. In the first 
stage, the probability that each student participates in the TNGTI program is modeled on 
available observable characteristics. By modeling selection into the program, this approach 
allowed us to compare participating and non-participating students who would have had a similar 
propensity to select into the program based on observables. In the second stage, the predicted 
probability of participation was used to model student outcomes. 

Stage 1: Creation of the Comparison Group. The outcome of interest in modeling propensity 
scores is treatment status ( 1 for students participating in the TNGTI program, 0 for the 
comparison group). To account for this binary outcome, logistic regression was used to model 
the logit (or log-odds) of student group assignment status. Because characteristics of students, 
the middle schools they attended, and the high schools they would attend influence whether they 
attend the TNGTI program, data on all of these pre -treatment characteristics were acquired from 
the Texas Education Agency. Student-level variables used to fit the propensity score models 
included: age, gender, race/ethnicity, special education, limited English proficient status, gifted 
education status, previous retention, number of 8 th grade disciplinary incidences, percent of days absent 
during 8 th grade, Texas Assessment of Knowledge and Skills (TAKS) scores in grades 4 through 8, and 
whether a student was classified as economically disadvantaged. Middle and high school 
characteristics used to fit the propensity score model included: percent race/ethnicity, percent LEP, 
percent special education, percent economically disadvantaged, percent bilingual, number of students, 
teacher/student ratio, number of full time teachers, accountability status, attendance rate, teachers’ 
average years of experience, mobility rate, and percent of teachers teaching subjects for which they do not 
hold a certification. Data were not available for each of these covariates for all students. To 
account for this, indicator variables were used to model the relationship between the pattern of 
missing data and propensity to participate in the program (Rosenbaum & Rubin, 1984). 

All pre-treatment covariates were initially considered as candidates for inclusion in the 
propensity score model. To select an initial propensity score model, we began by regressing 
each of the covariates on its own on program participation. All covariates with a p value of 
greater than 0.2 were then included in a forward stepwise regression function to produce an 
initial propensity score model. Propensity scores and propensity score logits were then estimated 
using this model. We examined overlap in the treatment and comparison groups and deleted non- 
overlapping cases. We then looked at balance across the two groups on all covariates. Balance 
statistics (standardized mean differences and variance ratios) were used to guide model selection. 
The final model included 79 covariates and the adjusted standardized mean differences between 
the treatment and comparison groups were below 0.2 on all pretreatment covariates, consistent 
with current best practice in the propensity score literature (Ho, Imai, King, & Stuart, 2007). 

Stage 2: Statistical Modeling of Student Outcomes. Outcomes of students in the TNGTI 
program were then compared with the outcomes of students who did not participate (the 
comparison group). We balanced pretreatment group differences in observed covariates using a 
propensity score stratification and marginal mean weighting approach (Hong & Hong, 2009). 
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Fifteen strata were used based on the spread and overlap of the data. The propensity score logit 
along with the pre-treatment eighth grade measure of the outcome were also included in the 
outcome model to control for within strata differences and residual bias (Schafer & Kang, 2008). 
Student outcomes were modeled using two-level hierarchical linear models to account for the 
nested nature of the data (students within high schools) as follows: 

Level 1 - Students 



IS 



= 



Poj + PijTNGTItj + ^ <*P s L sij + P 3} LP i} + P 4j Pretest^ 



S= 2 



Where y i; are the student level outcomes (9 th grade TAKS scores, 9 th grade attendance), TNGTI 
is an indicator of whether the student participated in the TNGTI summer program, L s y is an 
indicator variable for the logit propensity score strata, LP.. is the logit propensity score, and 
Pretestjj is the pre-treatment eighth grade measure of the outcome. 



Level 2 - High School 



Poj = Too + u oj 
Pij = Yio 
@2j = Y20 

p3j = 730 
p4j = 740 



In addition to modeling both ninth grade attendance and ninth grade TAKS scores as an 
outcome, the study examined whether attendance was a mediator for the program effect on ninth 
grade TAKS scores. That is, we investigated the theory that participating in the summer program 
led to increased attendance in ninth grade, which in turn led to increased learning reflected in 
ninth grade TAKS scores. These analyses followed the guidance outlined in Schochet (2009). 
While it should be acknowledge that they analyses are exploratory in nature and should not be 
interpreted causally, the evaluation team thought they provided valuable insights as we try to 
better understand program mechanisms and get within the proverbial “black box.” 



Findings / Results: 

Description of the main findings with specific details. 



All of the analyses described above have been completed. However, the evaluation report is 
currently with the Texas Education Agency for review. Until this review is complete, findings 
cannot be shared publically. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 



This study provides an important examination of the effects of providing additional supports to 
at-risk students as they transition into high school. Given the critical role ninth grade plays in 
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students’ success in high school and beyond, this information will undoubtedly be valuable to 
education practitioners and policy makers alike. 

In addition, the study is a strong example of the use of propensity scores to estimate the 
effectiveness of a program when randomization was not feasible. As in any observational study, 
it is not possible empirically validate that the strong ignorability assumption holds in this case. It 
remains possible that unobserved differences between students who participated in the TNGTI 
program are confounding treatment effect estimates. However, the rich array of pre-treatment 
covariates available in the study, including multiple years of pretest data, give reason for 
confidence in the inferences drawn from this study. 
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